Digital camera that uses object detection information at the time of shooting for processing image data after acquistion of an image

ABSTRACT

A digital camera includes an object detecting unit which analyzes an analysis target image including a through image data taken in from an image pickup device to compose a shot of an object, a shot image, or a reduced image, based on a feature appearing in an image representing a subject of shooting, and detects at least one main object captured in the analysis target image, a subject information extracting unit which extracts, for each image of object detected by the object detecting unit, subject information including a position in a screen corresponding to the analysis target image and its expansion, and a tag data writing unit which writes the subject information, as a part of tag data corresponding to the shot image, to a storage medium, thereby enabling use of the information acquired by object detection at the time of shooting, also after the acquisition of the image.

This is a Division of application Ser. No. 11/905,587 filed Oct. 2,2007. The disclosure of the prior application is hereby incorporated byreference herein in its entirety.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2006-279248, filed on Oct. 12, 2006, theentire contents of which are incorporated herein by reference.

BACKGROUND

The present invention relates to a digital camera having facilitieswhich detect a main object such as a person from through image data,that is, image data obtained by thinning-out reading from an imagepickup device to provide a moving image for view finder to a user, andprovide information related to the detected object for a process torealize shooting support facility such as automatic focusing (AF)control.

DESCRIPTION OF THE RELATED ART

As shooting support facilities for a digital camera a digital camera, atechnique of controlling an automatic focusing (AF) control mechanism onthe basis of an image obtained from a preset expected subject area or anarea designated by the user operating a button or the like provided onan operating panel is proposed.

Moreover, a technique of analyzing through image data acquired in theprocess of composing a shot by the user before shooting and therebydetecting an area having a feature of an image representing a humanface, and focusing on the human face detected as a subject on the basisof image data of the detected area, is proposed and practically used.

By using this face detecting technique, it is possible to securely focuson a subject irrespective of whether there is a main object in a focusadjusting area provided in advance at the center of a frame or the like.Thus, shooting in a free composition can be realized.

Meanwhile, a technique of extracting a featuring point of a human facefrom a shot image and identifying an individual face obtained as asubject in the shot image on the basis of the degree of matching with afeaturing point of a registered person, is proposed (Japanese UnexaminedPatent Application Publication No. Hei-6-259534).

Note that, as image pickup devices provided in digital cameras have anincreasing number of pixels, the image quality of shot images issignificantly improved. For example, even when a part of a shot image isenlarged in playback, the image has sufficiently high quality that canbe viewed clearly.

As one of the facilities to utilize such good image quality of thedigital camera, a facility is provided which enables a liquid crystaldisplay section provided in the digital camera or a projector connectedto the digital camera via a cable to enlarge a desired part of a shotimage in display.

To deal with such enlarged display facilities, in the digital camera inwhich AF control is carried out on the basis of an image in an expectedsubject area or a user-designated area as described above, informationindicating the expected subject area or the designated area is recordedto a storage medium together with the shot static image (hereinafterreferred to as a shot image), a thumbnail image corresponding to thisshot image, tag data related to the shooting date and time and so on(Japanese Unexamined Patent Application Publication No. 2004-193977 andNo. 2003-125251). As the information related to the expected subjectarea or the designated area is thus recorded in association with theshot image as a part of the information related to the shooting, linkingto the enlarged display facilities after shooting is realized.

Meanwhile, the above-described technique of Japanese Unexamined PatentApplication Publication No. 2004-193977 assumes that a main object iscaptured in the expected subject area, as an image to be shot.Therefore, for example, it cannot deal with a case of capturing theobject in the expected subject area, then half-pressing the shutterbutton to focus, and then shooting an off-center subject. The techniqueof Japanese Unexamined Patent Application Publication No. 2003-125251enables execution of AF control for a designated area located at adesired position. To this end, however, the user must operate the AFcursor to decide the designated area during shooting.

On the other hand, in the digital camera employing the face detectingtechnique, secure AF control can be carried out wherever in a frame theface of a person is located. Therefore, satisfactory shooting can becarried out with a composition having an extremely high degree offreedom and without performing complicated operations.

However, in the related art, the information acquired by face detectionis provided exclusively for a shooting process such as AF control. Nolinking is made with facilities prepared for the processes after theacquisition of the shot image (for example, slide-show facilitiesincluding the above-described enlarged display facility).

Note that, as the capacity of the storage medium for storing the imageis increased, the number of images shot by the user of the typicaldigital camera increases very much.

Therefore, for example, a technique of automatically filing a largenumber of shot images on the basis of various viewpoints, such as filingby each shot object, in addition to the shooting date and time, isdemanded.

For example, if the above-described face detecting and recognizingtechnique is applied to shot images, it is possible to extract images inwhich each object is captured. It is thus possible to selectively takeor leave shot images depending on whether a desired object is capturedin the images, and to file the shot images by each object.

However, since the image data shot by the recent digital camera is of anextremely large volume, as described above, detecting an areacorresponding to a human face in a shot image requires longer amount oftime corresponding to increasing the number of pixels in the shot image.

SUMMARY

It is an object of the invention to provide a digital camera in whichinformation acquired by object detection at the time shooting can beused also after acquisition of an image by the shooting.

It is another object of the invention to enable use of subjectinformation for object detection acquired in the shooting process, inslide-show with display of shot images or in filing works.

The above objects are achieved by a digital camera configured asfollows.

An object detecting unit analyzes an analysis target image including athrough image data taken in from an image pickup device at the time ofcomposing a shot, a shot image acquired by shooting, or a reduced imageacquired by reducing the shot image, on the basis of a feature appearingin an image representing a subject of shooting including a person, ananimal and a plant, and detects at least one main object captured in theanalysis target image. A subject information extracting unit extracts,for each image of at least one object detected by the object detectingunit from the analysis target image corresponding to a shot imageacquired by shooting, subject information including position data whichrepresents a position of the image of the object on a screencorresponding to said analysis target image and corresponding to anexpansion of the image of the object. A tag data writing unit writes thesubject information extracted corresponding to each object, as a part oftag data corresponding to the shot image, to a storage medium whichstores image data representing the shot image.

The above objects of the invention are also achieved by another digitalcamera configured as follows.

In the object detecting unit provided in the above-described digitalcamera, a face detecting unit detects, an area of the person's facecaptured in the analysis target image that occupies the screen. Thedetection is carried out on the basis of a feature appearing in an imagerepresenting a person's face. In the subject information extractingunit, a position data collecting unit collects position datarepresenting a position and size of the area detected by the facedetecting unit, as a part of the subject information. An alignment datacollecting unit collects alignment data representing alignment offeaturing parts including eyes and mouth in an image representing eachof at least one object face in each area detected by the face detectingunit, as a part of the subject information.

The above-described objects are also achieved by another digital cameraconfigured as follows.

In the subject information extracting unit provided in theabove-described digital camera, an area data collecting unit collectsinformation representing, by pixel, a pixel position in the analysistarget image corresponding to an upper left point in a rectangular areaincluding an image of the object detected by the object detecting unitand a size of the rectangular area, as a part of the subjectinformation.

The above-described objects are also achieved by another digital cameraconfigured as follows.

In the tag data writing unit provided in the above-described digitalcamera, a criteria storing unit stores a criterion registered in advanceto determine significance of at least one object detected by the objectdetecting unit. A significance evaluating unit evaluates significance ofeach object on the basis of the subject information extracted by thesubject information extracting unit and the criterion. A sorting unitsorts each subject information in accordance with the significance shownby a result of the evaluation by the significance evaluating unit, andprovides the sorted subject information for a writing process of the tagdata.

In the digital camera according to the invention configured as describedabove, it is possible to provide subject information related to anobject acquired at the time of shooting, for a process related to shotimage data.

Thus, in various facilities such as enlarged display in slide-showfacilities or image filing for each object, these facilities can berealized very quickly without a process of narrowing an area to beenlarged or an area where an object to be recognized exists.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view showing an embodiment of a digital camera according tothe invention;

FIG. 2 is a is a view showing a detailed configuration of each partrelated to production of tag data;

FIG. 3A and FIG. 3B are views showing an example of tag data;

FIG. 4 is a flowchart showing a tag data writing operation;

FIG. 5 is a view explaining a using method in slide-show facilities; and

FIG. 6 is a view showing a configuration of an image filing systemaccording to the invention.

DETAILED DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment of the invention will be described in detail.

FIG. 1 shows an embodiment of a digital camera according to theinvention.

In the digital camera shown in FIG. 1, light caused to form an image onan image pickup device 22 by an optical system for shooting 21 at thetime of shooting an image is converted to an electric signal accordingto its intensity by the image pickup device 22, then further convertedto digital data by an analog/digital (A/D) converter 23, and stored intoa memory 24.

Also, at the time of composing a shot before shooting, thinning-outreading is periodically carried out from the image pickup device 22(approximately 30-60 frames per second). This image signal is digitizedby the A/D converter 23 and provided via the memory 24 for aninterpolating process by an interpolating process section 31 of an imageprocess section 25. Thus, a so-called through image data is formed.

This through image data is sent to a liquid crystal display section (notshown) via a display process section (not shown), and it is provided fora process of providing a view to compose a shot to the user and alsoprovided for a process of detecting a person's face captured in thethrough image data, by a face detecting process section 32. Then, on thebasis of the result of the detection by the face detecting processsection 32, an automatic focusing (AF) control mechanism 29 iscontrolled by a shooting control section 28, and the optical system forshooting 21 automatically focuses on the person's face captured in thethrough image data.

Also, information extracted in the face detecting process by theabove-described face detecting process section 32 is sent to a subjectinformation collecting section 33 shown in FIG. 1. The information iscollected as subject information related to a feature of the face as anobject, as will be described later, and is sent to a tag data editingsection 34 via a subject information storing section 36.

A shooting instruction inputted by the user operating a shutter button(not shown) is transmitted to the image pickup device 22 and the imageprocess section 25 via the above-described shooting control section 28.In accordance with this, image signals corresponding to outputs of allthe pixels forming the image pickup device 22 are stored into the memory24. As these image signals are provided for the process by theinterpolating process section 31, image data representing one frame ofshooting image is provided. The image data acquired in this manner iscompressed by an image compressing process section 35 and is written toa storage medium 27 via a writing process section 26.

At this time, in response to an instruction from the shooting controlsection 28, for example, the subject information that is stored last inthe subject information storing section 36 is sent to the tag dataediting section 34 and provided for a tag data editing process, whichwill be described later. Then, the edited tag data is sent to thewriting process section 26 together with the image data compressed bythe image compressing process section 35 and is written to the storagemedium 27 such as an SD card.

Hereinafter, a facility to produce subject information and write it as apart of tag data in an exemplary case of using the result of facedetection will be described in detail.

FIG. 2 shows a detailed configuration of each section related toproduction of tag data. FIG. 3 shows an example of tag data. Moreover,FIG. 4 shows a flowchart showing a tag data writing operation.

In the subject information collecting section 33 shown in FIG. 2, aposition data collecting section 41 and a featuring data collectingsection 42 collect position data related to the position in the screenof each image representing the face detected in the process by the facedetecting process section 32 and featuring data related to the alignmentof eyes, nose, mouth and so on, which are featuring parts of the face,in the image corresponding to the detected face, respectively (step 301in FIG. 4).

The position data collecting section 41 can collect, for example, thecoordinates (X1, Y1) and (X2, Y2) of the upper left pixels of frames 1and 2 corresponding to the range of image corresponding to detectedfaces as indicated by bold broken lines in FIG. 3A, and the size (P1 byQ1, P2 by Q2) of these frames, as position data (see FIG. 3B). Also, thefeaturing data collecting section 42 can collect, for example, thecoordinate's (Ex1, Ey1) and (Nx1, Ny1) representing the positionsoccupied in the through image data by the images representing eyes,nose, mouth and so on acquired in the face detecting processcorresponding to the frames 1 and 2 shown in FIG. 3A, as featuring data(see FIG. 3B).

Also, in the subject information collecting section 33 shown in FIG. 2,a setting data collecting section 43 collects setting data includinginformation indicating whether AF control using the result of thedetection by the face detecting process section 32 is adopted by theshooting control section 28, and the size or the like of the throughimage data provided to the face detecting process section 32 (see FIG.3B). The setting data collecting section 43 sends the collected settingdata as a part of the subject information to the tag data editingsection 34.

The collecting process by the position data collecting section 41 andthe featuring data collecting section 42 is executed, for example, everytime the face detecting process is carried out by the face detectingprocess section 32, until shooting is carried out. The collectedposition data and featuring data are sequentially sent and stored intothe subject data storing section 36. In this case, as the subjectinformation storing section 36 overwrites the previous position data andfeaturing data with the newly received position data and featuring data,the up-to-date subject information is maintained in the subjectinformation storing section 36.

When a notification that a shooting process has been carried out is sentfrom the shooting control section 28 shown in FIG. 2 via an editingcontrol section 46 (affirmative judgment in step 302 in FIG. 4), asorting process section 44 provided in the tag data editing section 34shown in FIG. 2 sorts the subject information stored in the subjectinformation storing section 36 in accordance with a rule stored in apriority rule storing section 45 (step 303 in FIG. 4).

The priority rule storing section 45 shown in FIG. 2 stores, forexample, a rule that “it should be determined that significance ishigher for images of larger size detected as faces” and so on, as a ruleto determine the significance of each of the results of face detection.In this case, the sorting process section 44 refers to the position datacorresponding to each result of face detection stored in the subjectinformation storing section 36 and sorts these position data and thecorresponding featuring data in order from the largest size to smallersizes.

Also, in the case where face recognition based on the degree of matchingwith features of a registered face is carried out in the face detectingprocess section 32, it is possible to carry out a sorting process ofeach position data included in the subject information and thecorresponding featuring data on the basis that whether or not the resultof face recognition is included in the featuring data (see FIG. 3B).

In this manner, after the subject information is sorted in accordancewith the priority rule, a selective reading-out section 47, in responseto an instruction from the editing control section 46, reads out apredetermined volume of subject information from the subject informationstoring section 36 corresponding to the volume of the tag data inadvance (step 304 in FIG. 4), and sends the read-out data to a datacombining section 48. For example, after the selective reading-outsection 47 reads out setting data included in the subject information,the position data and the featuring data are read out in order from thehighest priority until the sum of the data volume of the read-outsubject information reaches a threshold value decided in accordance withthe volume of the above-described tag data. Thus, the position data andthe featuring data corresponding to significant objects can be providedfor the tag data editing process.

The subject information thus read out is sent to the data combiningsection 48 shown in FIG. 2. By this data combining section 48, thesubject information is added to shooting information (shooting date andtime, use or non-use of a flash, and so on) related to shooting controlprovided from the shooting control section 28. Thus, tag data as shownin FIG. 3B is formed (step 305 in FIG. 4).

In this manner, the tag data including the subject information relatedto the object captured in the shot image is produced. This tag data isassociated with the compressed image data provided to the writingprocess section 26 by the image compressing process section 35, and isthen written to the storage medium 27 (step 306).

Since the tag data including the subject information is written inassociation with the compressed image data corresponding to the shotimage as described above, the information related to face detection usedfor shooting control can be provided for the slide-show facility or fora post-shooting process such as an image filing process.

It is also possible to convert the position data collected by theposition data collecting section 41 shown in FIG. 2 as coordinatesrepresenting the pixel position in the through image data intocoordinates in a shot image formed by using the output of all the pixelsof the image pickup device 22, and then send it to the subjectinformation storing section 36, thereby omitting the similar coordinateconversion process at the time of using the slide-show facility.

It is also possible to carry out the above-described conversion processof coordinate values only for the position data read out by theselective reading-out section 47 and provide the coordinatesrepresenting the position in the shot image for the combing process withthe shooting information.

On the other hand, in the case where the size of through image datacoincides with the size of a thumbnail image, there is an advantage thatit is enabling the coordinate values representing result of facedetection in the through image, as direct use of the coordinate valuerepresenting the position of the face in the thumbnail image.

Also, the information to be written as subject information is notlimited to the results of detection acquired by the face detectingfacility, and it also possible to use the results of detection acquiredby a facility to detect a part of image data having a feature appearingin images of various subjects that can be considered to be objects, asimage data representing an image of an object, by analyzing throughimage data.

It is also posible to provide a shot image itself acquired by shootingor a reduced image such as a thumbnail image acquired by reducing theshot image, instead of through image data, for the process in the facedetecting process section 32, and write subject information includingthe result of face detection thus acquired, as a part of tag data.

Hereinafter, a method of using the above-described subject informationin the slide-show facility will be described.

In a slide-show, as shown in FIG. 5, compressed image data written inthe storage medium 27 such as an SD card is read out via a card reader51 in accordance with an instruction from a slide-show process section56. After the data is expanded by an image compressing andexpanding(C/E) process section 52, it is provided for the process in adisplay process section 54 via a memory 53 and is provided to the userby a display section 55.

For example, when the compressed image data is to be read out from thestorage medium 27, in accordance with an instruction from a slide-showcontrol section 62 shown in FIG. 5, the subject information included inthe corresponding tag data is read out by a subject informationreading-out section 61 and is sent to the slide-show control section 62.

The subject information, thus read out, is sent to a zooming processsection 64 via the slide-show control section 62. In response to this,the position data is converted on the basis of the ratio of the size ofthe through image data included in the subject information and the sizeof the shot image. An instruction to enlarge, in display, an area on thescreen indicated by the resulting coordinates and size is given to thedisplay process section 54. This enables realization of variousfacilities, for example, sequentially zooming each person's face in ashot image in which many persons are captured.

Here, since the above-described position data included in the subjectinformation shows the face position detected by the face detectingfacility in the through image data immediately before the acquisition ofthe shot image, the face position may not accurately coincide with theface position in the shot image. However, the composition of the throughimage data immediately before the acquisition of the shot imagesubstantially coincides with the composition of the shot image.Therefore, it suffices to adopt the face position in the through imagedata immediately before the acquisition of the shot image, for theprocess of enlarging a person's face in display as described above.Meanwhile, in recent digital cameras having a larger number of pixels,since the size of image data is extremely large, it tends to take a longtime to search image data representing a shot image for a partrepresenting a person's face. Therefore, it is very advantageous to usethe subject information acquired at the time of shooting as describedabove and thereby enable omission of such a search process.

In the case where the setting data included in the subject informationshows that the size of the thumbnail image and the size of the throughimage data coincide with each other, when reading the thumbnail imageone after another from the storage medium 27 and displaying them as anextemporaneous movie, the corresponding subject information is read outparallel to the reading-out of the thumbnail image, and the coordinatesin the through image data acquired as position data corresponding toeach result of face detection are directly sent to an extemporaneousmovie replaying section 65. Thus, they can be used for highlighting offace position in extemporaneous movie replaying facilities (pictmotionfacilities) using the thumbnail image.

It is also possible to use the subject information written in theabove-described manner, in an image filing system.

FIG. 6 shows a configuration of an image filing system according to theinvention.

Of the components shown in FIG. 6, the parts equivalent to those shownin FIG. 5 are denoted by the reference numerals shown in FIG. 5 and willnot be described further in detail.

In the filing processing section 57 of the image filing system shown inFIG. 6, subject information read out by the subject informationreading-out section 61 along with the reading-out of compressed imagedata is sent to a face recognizing process section 67 via a filingcontrol section 66 and is provided for a process of reading image datacorresponding to a face part of an object from the memory 53.

This subject information includes position data indicating the positionof the face detected by the face detecting facility. Therefore, as theimage data corresponding to the face part of the object is read out fromthe memory 53 on the basis of the position data, the process ofspecifying an area to which the face recognizing process is to beapplied can be omitted in the face recognizing process section 67 andthe time required for the face recognizing process can be reduced.

Although the face position indicated by the position data included inthe subject information may not accurately coincide with the faceposition in the shot image as described above, this lack of coincidencewill not cause any problem in narrowing the area to which the facerecognizing process is to be applied.

Meanwhile, the process of newly searching large-sized image data (forexample, image data of 8 million pixels) representing a shot image foran area where face recognition should be carried out, is verytime-consuming. Therefore, it is very advantageous that this searchprocess can be eliminated, and the face recognizing process by the facerecognizing process section 67 can be made significantly faster.Moreover, by using featuring data included in the subject information,it is possible to realize the process of detecting a main featuring partat a higher speed.

As the image data representing the shot image is classified by thefiling control section 66 on the basis of the result of face recognitionacquired in this manner, the image data can be stored, for example, intoa folder prepared in an image file storage 58 for each individual forwhich a featuring value has been registered in a database for facerecognition.

By thus using the subject information acquired at the time of shooting,it is possible to realize various filing processes such as classifyingand filing image data for each object, within a practical process time.

In the case where the result of the face recognizing process acquired bythe analyzing process for through image data at the time of shooting isincluded in the subject information, as shown in FIG. 3B, the result ofthe recognition included in the subject information may be directly usedas the result of the recognition by the face recognizing process section67. Alternatively, if the analyzing process is newly carried out for thecorresponding area of the shot image stored in the memory 53, theaccuracy of face recognition can be improved.

As described above, in the digital camera according to the invention,subject information related to an object acquired at the time ofshooting can be provided for the use in processes related to shot imagedata after shooting is completed.

For example, if the slide-show facility is linked, various playbackfacilities can be realized, such as enlarging and displaying pluralobjects one after another captured in one shot image. Also, if the imagefiling process is linked, for example, a target area can be limited inthe face recognizing process carried out for determination of an object,and the time required for the face recognizing process can besignificantly reduced.

Thus, in various facilities, for example, in enlarged display in theslide-show facility and image filing for each object, the process ofnarrowing an area to be enlarged or an area where the object to berecognized exists, is not necessary, and these facilities can berealized very fast.

Considering that the quality of images shot by using digital cameras hasimproved year by year and that their applications are greatly expanding,the features of the present invention that enable linking the facilityat the time of shooting with the display facilities and the filingfacilities after shooting are extremely advantageous not only in digitalcameras but also in various video information fields.

The many features and advantages of the embodiments are apparent fromthe detailed specification and, thus, it is intended by the appendedclaims to cover all such features and advantages of the embodiments thatfall within the true spirit and scope thereof. Further, since numerousmodifications and changes will readily occur to those skilled in theart, it is not desired to limit the inventive embodiments to the exactconstruction and operation illustrated and described, and accordinglyall suitable modifications and equivalents may be resorted to, fallingwithin the scope thereof.

1. A digital camera comprising: a characterizing portion detecting unitwhich analyzes an analysis target image including one of a through imagedata sequentially taken in from an image pickup device to compose one ofa shot of an object, a shot image acquired by shooting, and a reducedimage acquired by reducing the shot image, on the basis of a featureappearing in an image representing a subject of shooting including aperson and an animal, and detects one or more characterizing portions ofan object captured in the analysis target image; a subject informationextracting unit which extracts, for respective characterizing portionsof the object detected by the characterizing portion detecting unit fromthe analysis target image corresponding to a shot image acquired byshooting, subject information including position data which represents aposition of the characterizing portion of the object on a screencorresponding to the analysis target image and corresponding to anexpansion of the characterizing portion of the object; a positioningunit which specifies a position of the characterizing portion in theshot image by the position data of the subject information; and adisplay controlling unit which sequentially enlarges and displays eachof the characterizing portions in the shot image specified by thepositioning unit on a display section when the characterizing portiondetecting unit detects more than one characterizing portion.
 2. Thedigital camera according to claim 1, further comprising a tag datawriting unit which writes the subject information extractedcorrespondingly to the each object, as a part of tag data correspondingto the shot image, to a storage medium which stores image datarepresenting the shot image.
 3. A digital camera comprising: an objectdetecting unit which analyzes an analysis target image including one ofa shot image acquired by shooting and a reduced image acquired byreducing the shot image, on the basis of a feature appearing in an imagerepresenting a subject of shooting including a person, an animal, and aplant, and detects at least one main object captured in the analysistarget image; a subject information extracting unit which extracts, forrespective image of at least one object detected by the object detectingunit from the analysis target image corresponding to a shot imageacquired by shooting, subject information including position data whichrepresents a position of the image of the object on a screencorresponding to the analysis target image and corresponding to anexpansion of the image of the object; a tag data writing unit whichwrites the subject information extracted correspondingly to the eachobject, as a part of tag data corresponding to the shot image, to astorage medium which stores image data representing the shot image,wherein the tag data writing unit has: an evaluating unit whichevaluates a size of at least one object detected by the object detectingunit, and a writing controlling unit which writes tag data by sortingthe each subject information in accordance with the size of the at leastone object evaluated by the evaluating unit.